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(57) Abstract 

A congestion control scheme for a connection oriented packet network is disclosed. The congestion control scheme elimi- 
nates bufTer overflow and provides congestion free communication by eliminating the clustering of packets. Clustering is elimi- 
nated by imposing a smoothness requirement on the packet stream of each connection at the corresponding source node in the 
packet network. The smoothness is maintained for each connection over cveiy link traversed by a connection through use of a un- 
ique queuing strategy at the nodes wherein packets arriving at a node in an incoming frame are delayed and transmitted from the 
node in an adjacent outgoing frame. 



See back of page 



DESIGNATIONS OF "DE" 

Until further notice, any designation of "DE* in any international application 
wtiose international fHing date is prior to October 3. 1990, shall have effect in the 
territory of the Federal Republic of Germany with the exception of the territory of the 
former German Democratic Republic. 



Fm THE PURPmES OF INFOmATlON ONLY 

Codes used to identify States party to the PCT on the front pages of pamphlets publishing international 
applications under the PCT. 



AT 






ES 




MG 


Madafascar 


AU 






PI 


naked 


ML 


Mas 


B8 


BulMdos 




FR 


FnuBce 


MR 


Mauritania 


BE 






GA 


Gaboa 


MW 


Makwt 


BF 






GB 


UokedKafdom 


NL 


Netberlaods 


BG 


Bu%Brii 




HU 


tiunauy 


NO 


Normy 


BJ 


BewD 




rr 


Italy 


RO 


Romania 


BR 






jp 


iapao 


9> 


Sudan 


CA 


CkMda 




KP 


Dcatocntie Peopfe'St Rcpufalie 


S£ 


Sweden 


CF 


Cesna] Afiieu) Repu 


bic 




of Korea 


SN 


Senegal 


CC 


CongD 




KR 


RepubicoflCocca 


SJ 


Soviet Union 


CH 


SwMmhod 




U 




TO 


Chad 


CM 






uc 


SriLaaU 


TG 


Ta»> 


OE 


Gcmy. Men] Rcj 


pubic or 


LU 




IB 


United Sates or 


nc 






MC 









PCr/US89/04197 

1- 

CONGESnON FREE PACKET NETWORK 

FIELP QP THE IWEhmnNT 

The present invention relates to a traffic regulation and packet 
transmission scheme for connection-oriented packet networks which eliminates buffer 
5 overflow and provides congestion free communication. 

BACKOROTTrm OF THE mvEwxrnM 

A packet network generally comprises an array of switching nodes 

which are interconnected by transmission links. A typical connection involves the 

transmission of a packet stream from a source node to a destination node via one or more 
10 intermediate nodes and via one or more of the connecting links. The packet stream of each 

connection arrives at the corresponding source node from a packet source such as a unit of 

customer premises equipment. The packet stream is iUnstratively delivered to destination 

equipment after reception at the destination node. 

Congestion arises in a packet network for a variety of reasons. One 
IS way congestion arises is when sources of packets overload the network by transmitting too 

many packets into the network. Congestion also arises because particular sources are not 

smooth and transmit packets into the network in bursts. 

In a connection oriented packet switching network, congestion control 

is generally achieved by allocating, or reserving, transmission capacity for each connection 
20 on the appropriate links before the connection is setup. Such capacity allocation might be 

done based on the average rates of the connections (whereby statistical multiplexing is 

utilized) or based on the peak rates of the connections (whereby no statistical multiplexing is 

utilized). 

Neither the peak-rate capacity reservation approach nor the average- 
25 rate capacity reservation approach totally eliminates congestion in a packet switching 
network. 

The reason for this is packet clustering. Packet clustering arises in a 
packet switching network as follows. Typically, each node in a packet switching network 
utilizes a FIFO queuing strategy. This means that packets arriving at a node and destined 
30 for a specific outgoing link are queued and served by the outgoing link on a first-in, first- 
out basis. Sometimes, a packet from one particular connection is queued while packets from 
one or more other connections are served by the particular outgoing link. Very often, this 
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causes packets from the particular connection to duster together. This clustering 
phenomena can lead to congestion in a conventional packet switching network. When 
clustering occurs, buffers at the nodes may fill up. When the buffers at a node fill up, 
transmission to the node may be stopped resulting in congestion. If this happens at a 
5 plurality of adjacent nodes, packet network "gridlock" results. Alternatively, packets may 
be lost due to buffer overflow, depending on the size of the buffers. It should be noted that 
packet clusters can arise whether capacity is allocated based on the peak«rate or the 
average-rate of connections and can arise even if the packet streams of the connections are 
initially smooth. 

10 In view of the foregoing, it is an object of the present invention to 

provide a traffic regulation and packet transmission scheme for packets in a connection- 
oriented packet network which eliminates packet clustering so as to provide congestion free 
communication without packet losses due to overflowing buffers at the network nodes. 

STTVfVf AT^ V OF Xm? IKVENTTOK 

15 In accordance with the present invention, congestion in a packet 

network is eliminated by imposing a smoothness requirement on the packet stream of each 
connection as it enters the packet network at a source node, and by maintaining the 
smoothness (i.e. the absence of clustering) of the packet stream of each connection on each 
link traversed by a connection. The smoothness of the packet stream of each connection is 

20 maintained on each link traversed by a connection through use of a unique queuing strategy 
at the network nodes. 

To carry out the invention, a time period of duration T is globally 
defined for the packet network. Packets of particular connections arrive at the 
corresponding source nodes in successive time intervals or frames of duration T. In addition 
25 packets are transmitted between network nodes via links in successive time intervals of 
duration T. 

Each connection k in the network is allocated a rate r^. The packet 
stream of a connection k is smooth if the number of packets transmitted by the connection k 
in any frame of duration T does not exceed r^.T. The rates rj^, which are allocated to 

30 particular connections k, are chosen such that the packet stream of each connection is 

smooth when it arrives at the corresponding source node, i.e. the number of packets from a 
connection k arriving at the source node for connection k in any frame from a sequence of 
frames of duration T is bound by r^.T. 

This smoothness is maintained for each connection such as the 

35 connection k over all the links traversed by the connection through use of the following 
queuing strategy at the network nodes. 
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Packets arrive at a node via particular incoming links in incoming 
frames of duration T. Packets leave the node via particular outgoing links in outgoing 
frames of duration T. As used herein, an outgoing frame of a particular outgoing imfe is 
adjacent to an incoming frame, if it is the first outgoing frame which starts after the end of 
5 the incoming frame. Packets arriving at a node in an incoming frame and addressed to a 
particular outgoing link are delayed at the node and transmitted via the particular outgoing 
link in the outgoing frame which is adjacent to the incoming frame during which the packet 
arrived. 

Thus, if the packet stream of a connection is smooth as it enters the 
10 network at its source node and the queuing strategy described above is practiced at the 
network nodes, the packet stream is maintained as smooth over all links traversed by the 
connection, and packet clustering is eliminated. The primary cost involved in eliminating 
the packet clusters is the additional delay incurred at the nodes so that the packets are 
transmitted in the adjacent outgoing frames. This delay is not detrimental when an 
15 appropriate value for the duration T is chosen. No such delay is introduced when a 
conventional FIFO queuing strategy is used at the network nodes, and the result may be 
packet congestion. 

It is a significant advantage of the present invention that only a slight 
modification of the conventional FIFO queuing structure used at the packet network nodes is 
20 necessary to implement the smoothness-maintaining queuing stratngy. 

It is also a significant advantage of the present invention that the end- 
to-end delay for the packets of a particular connection are constant except for a small delay 
jitter. In contrast, when the conventional FIFO queuing strategy is used at the network 
nodes, the end-to-end delays for the packets of a particular connection are variable and 
25 unpredictable. 

The congestion control technique of the present invention eliminates 
buffer overflow and provides connection free communication for connection oriented 
services. The only other congestion control scheme for accomplishing this is a 
generalization of the time division multiplexing (TDM) concept and requires that time slots 
30 be assigned a priori for the transmission of packets for each connection (see e.g., Mukherji, 
U., 1986, "A Schedule-Based Approach for Flow Control in Data Communication 
Networks'*, Ph.D Thesis, MIT, Department of Electrical Engineering and Computer Science, 
Cambridge, Mass.). The present invention enables congestion free communication without 
the use of a slot reservation scheme. 
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jpp ppgntTPTTOW A WTNG 

FIG 1 schcmaticaUy Ulustratcs a packet stream which is smooth in 
accordance with the definition of smoothness utilized in the present invention. 

FIG 2 schcmaticaUy iUustratcs a packet network which can transmit 
5 packets in accordance with the present invention. 

FIGs 3a-3m schcmaticaUy Ulustrate the transmission of packets in 
accordance with a conventional FIFO queuing strategy and in accordance with the present 
iiiveation. 

FIG« 4a and 4b iUiutratB a node n and an arrangement of frames of 
10 duration T entering and leaving tiie node n.re»pectively. 

PIGa 5 and 6 iUustrate an alternative node n' and an alternative 
arrangement of frames of duration T entering and leaving the node n'. respectively. 

FIGs 7, 8, and 9 schematicaUy illnstrate node structures for 
transmitting packets in accordance with Utastrative embodiment, of the present invention. 
15 FIGs 10 and 11 illustrate a node n" and asynchronous outgoing frames 

departing the node n", respectively. 

FIGs 12 and 13 illnstrate an implementation of the node n" of TIG 10 
when the outgoing frame, are asynchronous as shown in FIG 11. in accordant with an 

illustrative embodiment of the present invention. 
20 FIG 14 illustrates asynchronous incoming frames arriving at a node n" 

of FIG 10. 

FIG 15 Ulustrate. a node structure for transmitting packets when the 
incoming frame, are asynchronou. a. .hown in FIG 14. in accordance with an iUu.trative 
embodiment of the present invention. 
25 FIGs 16 and 17 Ulnatrate an alternative mechanism for transmitting 

packet, when the frame, arriving at or departing a node are asynchronous, in accordance 
with an illMtrative embodiment of the preaent invention. 

pBTAn.F.n p pgrntTPTTON 

The detailed description of the invention i. divided into the foUowing 

30 .ections: Section a provide, a definition of smoothness for a packet stream. Section b 

shows how clustering arise when a conventional FIFO queuing strategy is used at the nodes 
of a packet network. Section i; set, forth the packet transmission scheme of the pre«nt 
invention including a unique queuing strategy at the network nodes and show, how th« 
transmission scheme eliminates packet dustering. Section i describe, how the network 

35 nodes are implemented to carry out the packet tran«ni..ion «*e»e of the pre.ent invention. 
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a. Definition qf Smoothness 

Before describing the packet transmission scheme of die present 
invention in detail, it is useful to explain more fully how congestion arises in a packet 
switching network. 

5 To do this, it is first desirable to define smoothness with respect to a 

stream of packets. A stream of packets is defined to be smooth with respect to the rate r (in 
packets per unit time) and the duration T, if the total number of packets arriving at a node 
over any interval [(m-l)T,mT] m -»...0,l...oo u bounded by r.T. Figure 1 illustrates a 
packet stream that is smooth with respect to the rate r=2/T and duration T. In FIG 1, the 

10 arrival of a packet at a particular node is indicated by an arrow. Thus» in any of the 

intervals [0,Tl, [T,2T], [2T,3Tl, [3T,4T], [4T,5T] there is a maximum of two (i.e. r=2/r 
producted with T) packets. It should be noted that the definition of smoothness is 
dependent on the time origin specified. If in FIG 1, the origin is shifted by T/2 so that the 
intervals [T/2, 3T/2], [3T/2, 5T/2]. [5T/2, 7T/2] are considered, the stream is no longer 

15 smooth with respect to the rate r=2/T as some of the intervals have three packets. 

b. The FIFO Queuing Strategy Results In The Clustering of Packets 

To understand how packet clustering arises in a network, consider the 
packet network 10 of FIG 2. The network n of FIG 2 has five nodes, n=l, 2, 3, 4, and 5. 
The network 10 also has four interconnecting links L= 1,2,3,4. The capacities of the links 

20 L= 1,2,3,4 are C^-^^C^^ZIT^ C3=l/T, C^=5/T. 

The following connections are setup in the network of FIG 2. 
Connection k=l connects the source node n=l to the destination node n=4, connection k=2 
connects the source node n«l to the destination node n=2, connection k=3 connects the 
source node to the destination node n=3, and connection k=4 connects the source node 

25 n- 5 to the destination node n» 2. The transmission rates allocated to the connections 
k- 1,2,3,4 are r^=r2=r^=l/T and r^-4/T. These rates are feasible in that the sum of the 
allocated rates of all the connections traversing a link is equal to or less than the capacity of 
the link. 

FIG 3a shows a time scale used in connection with the network 10 of 
30 FIG 2. The time scale is divided into a sequence of intervals or frames of duration T. Fig 
3b shows the packets which arrive at the source node n= 1 and are to be transmitted via the 
connection k>= 1 to the destination node n=4. It should be noted that the packet stream of 
FIG 3b is smooth with respect to the rate r^= 1/T and the duration T in that no more than 
r^.T (i.e. one) packets arrives at the node n= 1 during each frame of duration T. 
35 FIG 3c shows the stream of packets arriving at the source node n = 1 to 

be transmitted to the destination node n»2 via the connection k= 2. The packet stream of 
FIG 3c is smooth with respect to r^^ 1/T and duration T in that no more than r^.T (i.e. 
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one) packet arrive, at the node n- lin any frame of duration T. 

FIG 3d shows the stream of packets arriving at the source node n=5 to 
be transmitted to the destination node n=3 vU the eonnecdon k=3. The packet stream of 
FIG 3d is smooth with respect to r3= l^T and duration T in that no more d>an TyT (i.e. 
5 one) packet arrives at the node n-5 in any frame of duration T. 

FIG 3e shows the stream of packet, arriving at the source node n- 5 to 
be transmitted via the connection k- 4 to the destination node n=2. Note that the packets 
of FIG 3e arrive at the node n-5 in bursts. Packet. 1-4 arrive one after the other «>d then 
there is a long gap before the next burst of packets, comprising packet. 5-8. arrive.. 
10 However, according to the definition of smoothneu used herein, the stream of FIG 3e « 
smooth. The number of p«dte.. arriving in any fr«ne of duration T is equal to or less than 

the product of r^- 4rt and T. 

Thus, all of the packet streams entering the network 10 of FIG 2 are 
i^tiaUy smooth. In addition, in any interval of duration T. the sum of the average rate, of 
15 the connections traverring any link U equal to or less than the cap«dty of the link. 

However, despite this, when a FIFO queuing strategy is used at the nodes, congestion 

develops in the network. 

FIGS 3f-3i show that it is the interaction of the packets from the 
various connections at the node, which ca»«» clu.teri«g and the resnltant congestion. In 
20 FIG. 3f-3i it i. a»umed that the propagation delays of the links and the processing delays at 
the nodes are negUgible. FIG 3f ri.ows the packet, of connection k= 1 as they arrive at the 
node n=2 via the link L- 1. To understand FIG 3f. it is necessary to consider FIGs 3b and 
3c The link L- 1 services packett arriving at the node n- 1 using the FIFO queuing 
strategy. A. shown in FIG 3c. the first packet arriving at the node n- 1 is the packet 1 of 
25 connection k-2. As shown in FIG 3b. the second packet arriving at the node n- 1 « the 
packet 1 of connection k= 1. Thus, in accordance with the FIFO queuing strategy, the 
packet 1 of connection k=l is queued while the packet 1 of connection k=2 is serviced by 
the link L=l. The third packet arriving at the node n=l i. the packet 2 of comiection k-1. 
Thus, the packet 2 of connection k-1 doe. not h«re to wmt for a packet from connection^ 
30 k-2 to be .erviced. Accordingly, a. .hown in FIG 3f the packet. 1 and 2 of connection k- 1 
arrive at the node n.2 relatively closer together than they previously arrived at node n- 1- 

As shown in FIG 3c the fourth packet to arrive at the node n= 1, is the 
packet 2 of connection k=2. The fifth packet to arrive at the node n-1 i. the packet 3 of 
connection k= 1. Thus, the packet 3 of eonnecdon k- 1 i. queued whUe the packet 2 of 
35 connection k-2 i. .erviced by the link L=l. The rixth packet to arrive at the node n-1 is 
U.e packet 4 of connection k-1. Since this packet does not compete with a packet from the 
com«ction k-2. it is u^viced immedi^ely by the link L-1. For this reason, the packets of 
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connection k= 1 arrive at the node ii= 2 in pairs i.e. packets 1, 2 of connection k= 1 are 
paired, packets 3, 4 of connection k=l are paired. However, as shown in HG 3f. the 
packet stream of connection k- 1 arriving at die node n=2 to stiU smooth. The number of 
packets in any frame of duration T is still bound by die product of rj= lA" and T (i.e. one). 
5 In a similar fashion, die packets of connecdons k= 3 and k= 4 compete 

for service by die link L=4 at die node n=5. As shown in FIG. 3d and 3e. die first four 
packets to arrive at die node n=5 are a burst of packets 1-4 of connecdon k=4. The fifdi 
packet to arrive at die node n=>4 to die packet 1 of connecdon k=2. In accordance widi die 
FIFO queuing strategy, die packet 1 of connection k-3 U queued at die node n=4 whUe die 
10 packets 1-4 of connection k=4 are serviced by die link L=4. Only dien to die packet 1 of 
connection k=3 serviced by die link L=4. The sixtii packet to arrive at die node n=4 to die 
packet 2 of die connection k=3. and dito packet to serviced immediately by die link 1-4. In 
short, because of die FIFO queuing strategy at die node n=4, die competition for service 
between die packets of connecdons k=3 and k=4, and die lack of regularity in die arrival of 
15 die packets of connection k= 4, die packets of connection k= 3 arrive at die node 2 in 
clusters (see FIG 3g). In fact, die packet stream of FIG 3g to no longer smoodi. in diat 
some of the frames of duration T have more than TyT (i.e. one) packet. 

At the node n-2. the arriving stream of connecdon k- i (FIG 3f) and 
die arriving stream of connection k-3 (FIG 3g) compete for service by die link L=2 for 
20 transmission to node 3 using die FIFO strategy. Thus, for example, die packets 1,2,3.4 of 
connection k=l are serviced by die link L= 2 one after anodier but only after die packets 1 
and 2 of die connection k-3 are serviced. The result of dito competition on die stream of 
connection k- 1 to shown in FIG 3h. In particular, as shown in FIG 3h, die packets of 
connection k= I arrive at die node n-3 in clusters of four. Thus, to avoid congestion or 
25 packet overflow, a buffer for storing at least diree packets to needed at die node n= 3. If a 
smaller buffer to utilized, transmtosion to die node n=3 wiU be stopped or packets wUl be 
lost due to overflow. The stream of packets of connection k-1 arriving at die node n=3 to 
no longer smoodi. The number of packets arriving in some of die frames of duration T 
exceeds r^.T (i.e. one). An example of such a frame to die interval [4T.5T1. If a buffer of 
30 suitable lengdi to present at die node n=3. die packets of die connection k- 1 arrive at die 
node n-4 via die link L=3 in a smooth fashion as shown in FIG 3i. 

It should be noted that if more nodes and more connecdons are added 
to die network 10 more and more packets wiU duster togedier requiring larger and larger 
queues to prevent congestion or queue overflow. 
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e. The Packet Transmission Scheme ef the Present 

Invention EHmtnates Packet Clustering 

In accordance with the present invention, congestion and buffer 
overflow are eliminated by imposing a smoothness requirement on the packet .««n of «ch 
5 connection as it enters the corresponding source node. CThe packet .tr««« of FIG. 3b. 3c. 
3d and 3e of connection. k=1.2.3.4 meet flii. smoothness requirement). More parUcularly. 
ti.e number of packet. tran«nitted by connecdon k to the source node for connection k « 
bound by rj^.T. This smoothness requirement is maintained for the connection k on each 
link in the network traversed by the connection k. 

To insure that a connection whose packet stream is imnaUy smooth, 
and remains smooth on each link traversed by the connection, a unique queuing strategy is 
utilized at the network nodes. Packets, arriving at a node in an incoming frame of duration 
T and destined for outgoing link L. are delayed at the node, and transmitted from the 
node via the outgoing link L in an outgoing fcame of duration T. which is adjacent to the 
15 incoming frame during which the packet arrived. A. used herein, an outgoing frame on a 
particular outgoing link is ^ijacent to an incoming frmne. if it is the first outgoing frame on 
that outgoing link which starts after the incoming frame is complete. When the packet 
stream for each connection U initially smooth (i.e. when the number of packets arriving at 
the source node for connection k is bound by r^.T). this queuing straU^gy insures that the 
^0 packet stream of each connection remain, smooth on each link traversed by a connection 
{i.e. that on mxy link tr.ver.ed by the connection k the nmnber of packets m a frame 

duration T is bound by r^.T). . x « 

FIG 4b iUnstrates the frames 20 arriving at a node n (see FIG 4a) on 

an incoming link L' and the frames 30 leaving the node n via «. outgoing link L. Note that 
25 incoming and outgoing fr«ne. of HG 4 are entirely synchronous. The incoming and 

outgoing frame, can be so synchroni«d for aU nodes in a network when propagation delays 
and processing delay. « the nodes are negUgible. As indicated above, an outgoing frame 
and an incoming frame are adjacent if the outgoing frame is the first outgoing frame whxch 
starts after the completion of the incoming frame. Thus, in FIG 4. outgoing frame 31 « 
30 adjacent to incoming frame 21 «id outgoing frame 32 is adjacent to incoming frame 22. In 
accord«.ce with the present invention a p«*et arriving at the node n during the incommg 
frame 21. leaves tfate node n in the adjacent outgoing frame 31. 

FIG 6 iUustiates a situation where the incoming and outgoing frames 
at a node are not synchronized. A. shown in FIG 5. a node n' has two incoming links L'. 
35 L" and an outgoing link L. The incoming frmnes 40 arrive on the link L' mid the mcom.ng 
frames 50 arrive on the link L". The outgoing frames 60 leave the node n' on the outgomg 
Unk L. The outgoing fr««e 61 is adjacent to the incoming frame 41 on link V and the 
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i^co^ing fraoc 51 on Un. L". Si«U«rIy. the outgoing fra«. 62 i. adjac^t *c u.c.»n^ 
^a»c 42 on Unk L' and the Incoming frame 52 on link L'M- «xordancc ^th pmcnt 
invention, a packet P arriving on iink L' during the frame 42 leaves dur«g ^e adjacent 
:u«oingfr«ne62.Ii.a.imil.r,ashion.apacketP.arrivesonthe^^^^ 
5 Jmii irame 5t and leaves during the adjacent outgoing fr«ne 61. Thus the paclce« 

are not served by the outgoing link L in a first-in. iirst^t order. Inste«l. the packet P 
arrives at the node after the packet P but leave, the node first 

FIG. 3h3m riiow that when the node, utilize the queuing strategy of 
the present invention (rather th«. the conventional FIFO queuing strategy) in co^e.^on 
with the example of the network 10 of FIG 2. packet clustering is 

rbasedonliramesof durationTdefinedinFIGSa. I^okingfi^tatnCHSb 3^ 
packet 1 from connection k=2 and packet 1 from connection k-1 arrtve "-de 1 du^ the 
interval or frame [O.Tl. Thus both of these packet, are delayed at node n= 1 ^^e^ 
interval IT.2T, (i.e. the «lj«.nt outgoing frame) during which both are - ^« 

IS «>den=2via.heli«kL=l. The arrival of the p«*et 1 of connection k=l at node n 2 u 
showniuFIGai. Thepaeket20fconnectiouk-larrivesatthenoden=l^^^^^ 
interval [T.2T1. This packet i. delayed at the node n- 1 untU the mterval 2T.3^ (^e^the 
adiacent outgoing frame), during which interval it i. tr«»mi«ed to the node n-2. The 
arrival of the packet 2 of connection k= 1 at the node n-2 during the mterval t2T.3Tl « 
.0 IwninFIGSj. SimUarly. packet 3 of connection k=l and packet 2 of co^ecUon^^^^^^^ 
arrive at node n= 1 during the interval [2T.3T,. Theae packet, are ' 
n= 2 during the adjacent outgoing frame which i. [3T.4T1. The arrival of the packet 3 
connection k=l at node n=2Uriiown in nG3j. . , « .u. 

By comparing HG 3j with FIG 3f it can be seen that the amval of the 
25 packets of the connection k= 1 at node n=2 is much smoother when the queuing strategy of 
™t invention is used (FIG 3 j) than when the stmulard FIFO queuing strategy » used 

The tnm.mis.ion of packet, of connection k=3 and k=4 from node 
„=5 to the node n=2 in accordance with the present invention i. now considered. As 
hown in FIGS 3d and 3c. in the interval [O.T,. packet 1 of com.ec.ion k-3 and pack^ 1^ 
connection k= 4 arrive at the node n= 5. The« p-^ef «e transmined to ^e no e n 2 
.he adjacent outgoing fr«ne.i.e. the interval rr.2Tl The arrival of the pa<^« 

connection k=3 at node n=2 i. .hown in FIG 3k. The arrival of the remamder of the 

packet, from connection k=2 at the node n=2 is also shown in FIG 3k. 
packet. ^^^^ ^^^^ ^^^^^ 

from node n=5 to node n=2 in accordance with the present invention, the .tream of packets 
ofLectionk.3which arrive, at noden-2Umuch.moother than if the convenuonal 



30 
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FIFO queuing strategy is utilized. (Compare FIG 3k and FIG 3g). In fact, the packet stream 
of FIG 3g» which results when the packets of the connection k= 3 are transmitted from node 
n»5 to node n^2m accordance with the conventional FIFO queuing strategy » is not smooth 
(i.e. the r^.T limit on the number of packets per frame is violated in the intervals 
5 [T,2T],[5T,6T] etc.) In contrast, the stream of FIG 3k which results when the packets of 
connection k=3 are transmitted from node n^S to node nB2 in accordance with the present 
invention, is smooth. 



are serviced by the link L»2 for transmission to the node n«3. The arrival of the packets 

10 from connection k^ 1 at the node n=3 is shown in FIG 31. As shown in FIGs 3j and 3k, the 
packet 1 from connection k= 1 and the packet 1 from connection k=3 arrive at the node nB2 
during the interval [T,2T]. In accordance with the present invention, these packets are 
serviced in the adjacent outgoing frame, i.e. the interval [2T,3T]. Thus, as shown in FIG 
31, the packet 1 of connection k» 1 arrives at the node=3 during the interval [2T,3T]. 

15 Similarly, the packet 2 from connection k— 1 and the packet 2 from connection k»3 arrive at 
the node n»3 during the interval [2T,3T]. These packets are serviced during the interval 
[3T,4T] (i.e. the adjacent outgoing frame) so that as shown in FIG 31, the packet 2 of the 
connection k» 1 arrives at the node n»3 during the interval [3T,4T]. In this manner all the 
packets of connection k« 1 arrive at die node n»3. 

20 It should be noted that the stream of packets from the connection k= 1 

which arrives at the node n= 3 is smooth. As shown in FIG 31, no more than one packet of 
connection k= 1 per frame of duration T arrives at the node n=3. Thus, a buffer for storing 
one packet is all that is required at the node n=3. In contrast when the FIFO queuing 
strategy is used, tiie packets of the connection k= 1 arrive at the node n - 3 in clusters of 

25 four packets as shown in FIG 3g so that a buffer for storing three packets may be required 
to avoid packet loss. Thus, when the transmission scheme of the present invention is 
utilized, packet clustering is eliminated thereby eliminating any resultant congestion or 
buffer overflow. In particular circumstances, shorter buffers may be utilized. The 
transmission of the packets of connection k= 1 from node n=3 to node n=4 is via the link 

30 L" 3 shown in FIG 3m. 



smoothness property on all connections throughout the network. In accordance with the 
present invention packets may be viewed as being transmitted between nodes via links in 
sequences of time intervals or frames of duration T. Ilie duration T is globally established 
35 for the entire network. It should be noted that the time origin for the frames on different 
links may be chosen differently. For example, at a node all the arriving frames or all the 
departing frames may be synchronous. 



At the node 2, the packets from connection k= 1 and connection k=3 



In short, the present invention eliminates congestion by imposing a 
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me foUowing criteria may be considered in choosing the duration T. 
First, it should be noted that for a longer duration T. the packet, will experience greater 
end.to^d deUys. Thus. T should be chosen so that the resulting end-to^nd p«:ket deUys 
are tolerable for the wavices being offered. Second, it should be noted that for a longer 

5 duration T. the buffer requirements at the node, are greater, can be shown that the 
aaximmn buffer capacity for a link L is 3C^r where U the capacity of the link). Thus. 
T should be chosen so that the resulting buffer requirements are feasible. Thirdly. T should 
be chosen large enough to accommodate the «naUest rate allocation which is utilized (i.e. T 
is preferably chosen larger than 1/r^). to «ldltio«. It U desirable for T to be chosen large 

10 enough «, tiiat the rates aUocated to particular connections are a. distant a. possible from 

the peak rates of the connections. 

A rate is aUocated to a connection k such that 



All connections 
traversing link L 

where C, is the capacity of the link L in packett per unit time. 

The rates r,^ are also allocated such that the packet stream of every 
connection k is smood. when it arrives at tiie source node for ti.e connection k. 
Alternatively, such smootimess may be imposed by suppressing tiie transmission of some of 
20 tiie packets from particular «.«rce.. As Indicated above, a stream Is smooti. if ti.e number 
of packets arriving at ti»e «,«rce node in a frame of duration T does not exceed r^.T. The 
rate r^ is in between flie instanomeous peak-rate of tiie connection k and tiie long-term 
average rate of tiie connection k. As tiie duration T U made larger, flie rate r^ asymtoticaUy 
approaches tiie long term average rate for tiie connection k. 
25 Thus, tiie packet stream arriving at tiie corresponding source node for 

each connection is smootii. The packets are tr^ismltted tiirough tiie network such tiiat tills 
.moodiness Is maintained individuaUy for each connection over each Unk traversed by tiie 
connection. 

TO maintain tiie smootimess. a unique queuing strategy may be utihzed 
30 at tiie network nodes. A p«*ct which arrives at a node in an incoming i«mc is delayed at 
the node and serviced by tiie appropriate outgoing link in tiie adjacent outgoing frame. As 
indicated above an outgoing frame on a particular outgoing link is adjacent to an incommg 
frame when tiie outgoing frame is tiie first outgoing frame to start after tiie completion of 
the incoming frame. 
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When rate, ate aUocated and packets are transmitted in the foregoing 
manner packet elnatcring l» eliminated. Wi.en packet clustering is eliminated the resulting 
p,ob J of congestion and overflowing buffer, are also eliminated. In fact the present 
Mention guarantee, congction free tran.mi.sion with waller ,ue»e size. th«. are re,utred 
5 When the conventional FIFO queuing strategy U utilized, to contrast to the present 

Invention, with the conventional FIFO queuing strategy, it i. ^o.t impo«*le to ^aranue 
congction ^ee communication no matter what the capacity of the buffer, at the node., to 
addition, when the traffic regulation and packet control «heme of the present unrentaon » 
Tied the end-to.nd delay for the packet, of a particular connection i. eon.^t ««pUor 
10 a small jitter delay, to contra.*, the conventional FIFO queutog .trategy result, u. end-to- 
end delay, whidi are variable and unpredictable. 
d. Node Structure for Carrying Out the IweMlve 

Packet Transmission Scheme 

Packet .witching nodes which .ervice arriving packets m accordance 

15 with the present invention are easUy implemented. FIG 7 show, an embodiment of the node 
„of FIG^awhereintheincomingframeswhicharriveviathelinkl.' are.y-chronou.wxth 

outgoing frame, which leave vU .he Unk L'. Synchronou. incomtog «.d ongoing ramcs 
are .hown in FIG 4b. to reaUty a typical node .uch a. the node n of HG 7 has a plurality of 
incoming Itok. and a pluraUty of outgoing links. However, for purpo.e. of danty only one 
20 incoming link L' and one outgoing link L are .hown in FIG 7. 

20 mcommg ^^^^ ^ ^^^^^ ^ .^^^ ^^^^ ^ . 

particular outgoing link., depending, for example, on addre» header -"^"-^ 
beader of the packet. Packet. de.ttoed for the outgoing link L emerge from the swU^^ 
on line 101. These packet, are routed by a switch 103 to one of two FIFO queue. 105.107. 
25 The switch 103 is synchronized with the incoming «.d outgoing frame, so that du«.g 
alternate incoming frames, the packet, are routed to the alternate FIFO I"'- "^'^"^^ 
Thus, during one tocomtog fr«ne the p«*et. are .tored to the FIFO queue 105 and m the 
next frame they are .tored to the FIFO queue 107. The packets are connected to *e 
outgotog Itok L VU the .witch 109. The .witch 109 is out of phase with the sw.tch 10 su^ 
30 that dulg a frame to which packet, are betog .tored to the queue 105. packet. .»red m the 
queue 107 durtog the previou. frame are provided wlti» acces, to the outgomg Imk L. to 
L manner, a packet which arrive, to an tocomtog fr«ne i. served by the Itok L « the 
adjacent outgoing frame.^^^^ ^ ^^^^ ^ .^^^^ ^ „^ , 

35 Arriving packet, which are routed by the switch 100 to the outgotog Unk L. emerge from 
.he.witLl00onitoel01andarebufferedtotheFIFOquenel04. At t^ beg.nmng of 
each outgotog frame the service controUer 106 mark, the number of packet, u. the queue 



wo 90/11659 PCr/lS89/04197 

-13- 

104 which arrived in the just preceding arriving frame. (Note, since the Incoming and 
outgoing frames are synchronous, an outgoing frame begins Immediately at the end of the 
incoming frame). Under the control of the service controUer 106. the switch 111 connects 
the outgoing Unk L to the FIFO queue 104 at the beginning of the outgoing frame period 
5 and for just enough time so that only the packets which arrived in the queue 104 daring the 
immediately preceding arriving frame achieve access to the outgoing link L. In this manner 
arriving packets leave via the outgoing link L in the adjacent outgoing frame. For the 
remainder of the outgoing frame period, Ac ou«olng link L U Idle. 

It should be noted that the structure of the node of FIG 8 differs from 
10 the structure of a node in a conventional packet network which utilizes a FIFO queuing 
strategy only in that the service controUer 106 and switch 111 are included. It is thus a 
significant advantage of the transmission scheme of the present invention, that it can be 
implemented with only relatively minor modifications to the nodes of existing packet 
network. 

15 FIG 9 shows a node structure which enables the idle period of the link 

L to transmit low priority packets. lUustratively. the p«*et network which includes the 
node n may transmit high priority packets in accordance with the congestion control scheme 
of the present invention and lower priority packets in accordance with an alternative 
congestion control scheme. As shown in FIG 9. low and high priority packets emerge from 
20 the switch 100 on line 101. The multiplexer device 121 routes the high priority packets to 
the FIFO queue 104 and the low priority packets to the buffer 122. During the portion of 
each frame for which the switch 111 is not comiected to the FIFO queue 104. it is connected 
to the buffer 122 so that the low priority packets can access the outgoing link L. 

FIG 10 shows a node n" which has two incoming links L^'. and L'j 
25 and two outgoing links and 1^. The incoming frame, on the links L'j. and L^' arc 
shown in FIG 11 along with the outgoing frame, on the Unk. Lj. Lj. The incoming frames 
on the incoming links are synchronous. However, the outgoing frames on the outgoing 
links Lj. L- are asynchronous. The time period between the end of an incoming frame and 
the beginning of the adj«»>t outgoing frame on Unk Lj is 6^,. The time period between the 
30 end of an incoming frame md the beginning of the adjacent outgoing frame on Unk 1^ is 

To account for the time periods ex,, and e^., the node structure of FIG 
8 is modified to include appropriate delay elements. In particular, as shown in FIG 12, 
packets arriving on the Unks L^' «id 1^' and destined for the outgoing Lj. leave the switch 
35 100 via the Une 101 and are stored in the FIFO 104. The service controUer marks the 
packets which enter the FIFO 104 during each arriving frame period. The switch 111 
connects the outgoing Unk to the FIFO 104 at the end of each arriving frame period for 
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enough time for the packets which arrived to the preceding arriving frame period to 
^ the outgoing link L,. For the remainder of the time, the link U disconnected from 
U.e FIFO 104. To insure that the packets which leave the FIFO 104 are tnm«nit.ed vm the 
outgoing link L, in the adjacent outgoing frame, the p«dcet. are processed by the delay 
5 element 130 of duration before tnmsmission via the outgoing link 1^. A similar delay of 
duration 0., I. provided for ^ outgoing link I,. The delay elements 130 have the effect of 
synchronizing the outgoing frames with the incoming frames. 

In a preferred embodiment of the Invention, as shown in FIG 13. the 
Bode n" may be implemented by incorporating the delay 130 Into the FIFO 104. This 
10 enable, the node n" of FIG 13 to be implemenU:d with only a minimal modification of 
nodes in a conventional packet network. In this case, the service controUer 106 marks the 
number of packets which arrived in a just ending incoming frame 6.. seconds before the 
beginning of the next outgoing frame. At the start of the next outgoing frame, the hnk Lj 
is connected to the FIFO 104 for only enough time for the marked number of packets to be 
15 serviced by the link L,. This insures that each arriving packet is serviced in the adiaeent 
outgoing frame. The node of FIG 13 may be modified to include a bu«er for low prior.ty 

packets as shown in no 9. . „ , 

FIG 14 shows an alternative frame arrangement for the node n ot 
FIG 11. Recall that in the frame structure of FIG 10. the frames arriving on the incoming 
20 links L,'. Lj' are synchronous and the frames on the outgoing links 1^, I^ are 

asynchronous. In contrast, in FIG 14. the frames on the incoming links L^' . L^' are 
synchronous and the frames on the outgoing link. L,. are synchronous. The ume 
period e.i is the time period between the end of an «Tiving frame on incoming Unk and 
the beginning of the adjacent outgoing frame on the outgoing links L^. L^. Simdarly, 9,^ « 
25 the time period between the end of an arriving frame on incoming Unk I^' and the 
beginning of the adjacent outgoing frames on the links Lj, I^. 

A modification of the node of HG 8 to account for the time periods 
e,- m.d e.- is shown in FIG 15. Packets arriving on the incoming links Lj'. I^' are delayed 
by'the delly element. 140.150 of durations . H'. respectively. The delay elements have 
30 the effect of synchroniring the incoming frame, on link. L,', L^' with the outgoing frames 
so that packets depart the node of HG 15 in the adjacent outgoing frame. 

AS shown in FIG 15. after being processed by the delay elements 140, 
150 packets destined for the outgoing link I^. are routed by the switch 100 to the line 101 
and FIFO 104. The service controller 106 and switch 111 control transmission of packets 
35 from the FIFO 104 to the outgoing link L, In the manner described above with reference to 
FIGS. 
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Another possible frame anangement for the node n" of FIO 11 to for 
the arriving frames on the incoming links . Lj' to be asynduonom and for the outgoing 
frames on tbe ongoing links to be asynehronous. To Insure diat packets atiiving in 
particular incoming frames are transmitted in proper outgoing frames, appropriate delay 

5 elements (e.g. 140, 150 of FIG 15) are utilized at the interfaces between the incoming links 
and the nodes and appropriate delay elements (e.g. 130 of FIG 12) are utilized at the 
interfaces between the outgoing links and the nodes. Such delay elements have the effect of 
synchronizing the arriving and departing frames. In thto case, for some pairs of incoming 
and outgoing links, the packets of an incoming frame will be transferred to the outgoing 

10 frame immedtately after the adjacent outgoing frame. However, congestion free 

communication will still take place. 

An alternative scheme for handling packets arriving at a node in 
asynchronous incoming frames to iUustrated in connection with CTGs 16 and 17. FIG 16 
shows the frame arrangement for a node with three incoming links 1^' . Lj' . W 
15 outgoing link L^. The scheme HQs 16 and 17 to general and does not require synchronous 
outgoing frames. 

Consider a packet P arriving on incoming link L3' in Frame F3'. This 
packet leaves via the Uak in the adj»»nt outgoing frame which to Fj. The packet P may 
be delayed at the node by a maximum time period of just under 3T. If the packet P arrives 
20 at the beginning of the frame Ty and it leaves the node at the end of the frame Fj via the 
link L,. then the delay to approximately equal to the duration of the frame F3'. the duration 
of the mismatch e^;between the frame, on Lj' and the frames on Lj and the duration of the 
frame Fj. Since the duration of the mtomatch 9^; to always less than T. the maximum delay 
for the packet P to just under 3T. Now consider the packet P' which arrives via the link L^' 
25 at the end of the frame Fj'. The earliest thU packet can be transmitted to at the beginning 
of the adjacent outgoing frame F^. Thus, the packet P' to delayed at the node for a 
minimum time of e^.; which to the mtomatch between the frames on links 1^' and Lj. Since 
6^; can be quite small, the delay for the packet P' can approach zero. From the foregoing 
example, it can be seen that a packet to delayed at a node for a minimum time which can 
30 approach zero and for a maximum time which can approach 3T. Another way of looking at 
thto to to say all packets are delayed for one of three possible time ranges: 0 and T, T and 
2T. or 2T and 3T. Note also the packet P arrives at the node before the packet P' but leaves 

the node after the packet P' . 

A node structure for handling the frame arrangement of FIG 16 to 
35 iUustrated in FIG 17. The node 200 of FIG 17 include, a buffer 210 associated with each 
outgoing link .uch as the outgoing Unk L^. lUustratively, each buffer 210 to divided into 
three queues 211. The queues 211 of FIG 17 are served by the link for successive 
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outgoing frame interval, on a round robbin basis. Thus, for example, the queue 211 » 
.enrtced during the outgoing frame interval. [O.T]. 13T.4T1 etc (.ec FIG 16) and another of 
the queue. 211 >. wrviccd during the outgoing frame interval. [T,2TlI4T.5n etc 
Each packet arriving at the node 200 of HG 17 via one of the 

5 incominglink5L,^I^^ 

packet to indioue to which of the queue. 211 the packet U to be routed. The packets are 
then routed by the .witch 300 to the buffer unit 210 and stored in the appropriate queue 211 
depending on how long a delay i. required . In thi. manner it U a-ured that each p«ket i. 
delayed in the node by the appropriate amount «> it U tnm.mit.ed from the node in the 
outgoing frame which i. adjacent to it. incoming frame. 

A Mheme for transmitting packet, m a packet network mvolving 
traffic regulation at the «.«rce node, and a queuing ..rategy at the network nodes has been 
disclosed. The scheme eliminates congestion and buffer overflow by maintaining the packet 
stream of each connection as smooth over all the link, traversed by the connection «. a. to 
eliminate packet clustering. FmaUy. the above-dccribed embodiment, of the invention are 
Intended to be illustrative only. Numerou. alternative embodiment, may be devised by 
those .kUled in the art without departing from the .pirit and ««pe of the following dauns. 



10 
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1. A metiiod for transmitting packets in a packet network comprising 



What is claimed is: 
the steps of 

receiving packets at a node of said network on one or more incoming 
5 links in one or more sequences of incoming time intervals of duration T, 

transmitting each of said packett from said node on an outgoing link in 
a particular outgoing time interval from a sequence of ontgoing time intervals of duration T, 
the particular outgoing time interval for each packet being adjacent to the incoming time 
interval during which the packet arrived at said node, 

the maximum number of packett of a connection k arriving at said 
node or leaving said node in one of said time intervals of duration T being bound by v^.T. 
where r^ is a rate allocated to the connection k. 

2. The method of claim 1 wher^ said incoming and outgoing 

sequences of time intervals are synchronous. 
15 3. The method of claim 1 wherein said packett are received at said 

node on a plurality of incoming links and wherein the sequences of arriving time intervals 
associated with said incoming links are asynchronous. 

4. The method of claim 1 wherein s^d packett are transmitted from 
said node on a pluraUty of outgoing links and wherein the sequences of outgoing time 

20 intervals associated with said outgoing links are asynchronous. 

5. The method of claim 1 wherein the sum of the rates allocated to 
a plurality of connections k traversing a particular link in said network is less than or equal 

to the capacity of the link. 

6. The method of daim 1 wherein the rate r^^ allocated to a particular 
25 connection k is less than an instantaneous peak rate for said connection k and greater than a 

long term average rate for said connection k. 

7. The method of daim 1 wherein each connection k in said network 
has a source node and the packet stream for a connection k arriving at a source node in a 
frame of duration T is bound by r^.T. 

30 8. A method for transmitting packett in a packet switching network 

comprising a plurality of nodes interconnected by links, said method comprising transmitting 
packett between nodes via said links in sequences of time intervals of duration T. tiie 
number of packett being transmitted by a connection k in one of said intervals being bound 
by rj^.T for each link traversed by tiie connection k and for tiie entire duration of tiie 

35 connection k, r^^ being a rate aUocated to connection k. 
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9. The method of claim 8 wherein each packet arriving at a node vU a 
particular incoming link in a particular one of said time intervals and destined for a 
particular outgoing link U transmitted via said outgoing link in a time interval which is 
adjacent to the time interval in whidi the packet arrived at die node. 
5 10. The mefliod of «aaim 9 wherein the rate reallocated to each 

comiection k is less than the instantaneous peak rate of the connection k and greater than the 

long term average rate of the connection k. 

11. The method of daim 10 wherein the sum of the rates r^ allocated to 
aU the connections traversing a link is less than or equal to the capacity of the link. 

12. Themefliodof daim 8 wherrin the number of packets of each 

connection k arriving at a source node of connection k is bound by r^.T. 

13. The method of claim 8 wherein the time origins for the sequences of 
time intervals on different ones of said links is different. 

14. A metiiod for transmitting padtets in a padcet switching network 

15 comprising a plnraUty of nodes interconnected by links, said metiiod comprising transmitting 
. plurality of padcets between nodes via said link, in sequences of time interval, of duration 
T and delaying said packets at said nodes so ti.at each of said plurality of packets arriving at 
one of said node, via an incoming link in a particular one of said time intervals and destined 
for a particular outgoing link is trmumitted from said one node via said particular outgoing 

20 link in a time interval whidi begins at or after ti» end of tiie time interval in whidi tiie 

packet arrived at sud one node. 

15. A node for use in a packet network comprising 
means for receiving packets via one or more incoming links in one or 
more sequences of incoming time intervals of duration T, 
25 means for transmitting eadi of said padtets on an outgoing link in a 

particular outgoing time interval from a sequence of outgoing time intervals of duratibn T. 
said particular outgoing time interval being adjacent to tiie incoming time interval during 

which tiie padcet arrived at said node, 

for a connection k in said network, tiie maximum number of packets 
30 arriving at said node or leaving said node in one of said time intervals of duration T being 
bound by r^T where r^ U a rate allocated to tiie connection k. 

16. The node of daim 15 wherein said incoming time intervals and said 
outgoing time intervals arc synd^onous, and wherein said transmitting mean, indudes first 
and .econd FIFO queue. a.sodated witii said particular outgoing link, and switdi means for 
35 enabling padcet. to be stored into said first FIFO queue and transmitted out of said second 
FIFO onto said particular outgoing link and for enabling padcets to be stored in said second 
FIFO queue and transmitted out of said first FIFO onto said particular outgoing link in 
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altemate ones of said time intervals. 

17. The node of daim IS wherein said transmitting means includes a 
FIFO queue associated widi said particular outgoing link and control means for insuring that 
packets stored in said FIFO queue in one of said incoming time intervals are transmitted to 

5 said particular outgoing link in the adjacent outgoing time interval. 

18. The node of claim 17 wherein said control means comprises means 
assodated with said FIFO queue for marking the number of padcets arriving at said FIFO 
queue in one of said time intervals and switch means for connecting said FIFO queue to said 
particular outgoing link for a portion of the time interval which is adjacent to said one time 

10 interval so that said number of packets in said FIFO queue are transmitted to said particular 
outgoing link. 

19. The node of daim 18 wherein said packets are high priority packets 
and wherein a buffer for additional low priority packets is assodated with said particular 
outgoing link, said buffer being connected to the particular outgoing link when said FIFO 

15 queue is not connected tiiereto so that said low priority packets can be transmitted via said 
link. 

20. The node of daim 17 wherein said node receives packets on a 
plurality of incoming links whose sequences of incoming time intervals are asynchronous, 
each of said plurality of incoming links interfacing said node by means of a delay element. 

20 21. The node of claim 17 wherein packets leave said node in a plurality 

of outgoing links whose sequences of outgoing time intervals are asynchronous, each of said 
plurality of outgoing links interfacing said node by means of a delay element. 

22. The node of daim 21 wherein said FIFO queue indudes one of said 

delay elements. 

25 23. The node of claim 17 wherein said node receives packets on a 

pluraUty of incoming links whose sequences of time intervals are asynchronous, wherein 
said node further includes means for marking arriving packets to indicate how long said 
arriving packets are to be delayed at said node, and wherein said transmitting means 
indudes means for buffering said arriving packets for different durations as indicated by 
30 said marking means. 

24. A packet network comprising 
a plurality of nodes, 

a plurality of links interconnecting said nodes for transmitting a 
plurality of packets between said nodes in sequences of time intervak off duration T, and 
35 means assodated with said nodes for delaying said plurality of packets 

so that each of said plurality of packets arrivmg at one of said nodes via an incoming Hnk in 
a particular one of said time intervals is transmitted from said one node via a particular 
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outgoing 
interval. 



link in a time interval which begins at or after the end of said particular time 
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